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CLAIMS 



1 . A system that enables a user to interact with a virtual con- 
trol panel using a user controlled pointing object (1), wherein 
5 the system comprises a first tracking unit (15b) adapted to cap- 
ture data representing the position of the pointing object, char- 
acterized in that the system further comprises: 

- a portable identification element (3), 

- a second tracking unit (15a) adapted to capture data repre- 
10 senting the position of the Identification element, 

- a storage unit (29), storing at least one pre-defined graphi- 

cal interface representing a control panel of a device, 
said graphical interface comprising an interface for user 
interactions with the device, 
15 - a graphics unit (23), generating a graphical representation 

of the control panel based on said stored graphical inter- 
face, 

- a registering unit (25), registering said graphical represen- 

tation of the control panel in a fixed relation to said port- 
20 able identification element, based on said data repre- 

senting the position of the identification element, to pro- 
duce a virtual control panel, 

- a display unit (7), showing the user a view comprising the 

real world and the virtual control panel projected in a 
25 fixed relation to said portable identification element, and 

- an application unit (21), performing actions in response to 

the users interactions with the virtual control panel, and 
determining which actions to be performed based on the 
position of said user controlled pointing object in relation 
30 to the identification element. 



2. A system according to claim 1, characterized in t hat the 
system Is adapted to modify the appearance of the virtual con- 
trol panel In response to interactions between the user con- 
35 trolled pointing object and the virtual control panel. 
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3. A system according to claim 1 or 2, characterized in that 
said graphical interface is adapted to display data from the de- 
vice and that the system is adapted to generate a graphical rep- 
resentation of the data and to display the data on the virtual 

5 control panel. 

4. A system according to any of the previous claims, charac- 
terized in that said user controlled pointing object is a handheld 
pointing device (1) or a part of the user's body. 

10 

5. A system according to any of the previous claims, charac- 
terized in that the storage unit (29) Is adapted to store a plural- 
ity of graphical interfaces, each representing a control panel of 
a particular device, that the system is adapted to generate and 

15 display a plurality of graphical representations of control panels 
for different devices based on said stored graphical interfaces of 
the devices, and that the system comprises means (31) for iden- 
tifying which of the stored control panels to be displayed. 

20 6. A system according to claim 5, characterized In that said 
means for identifying which of the stored control panels to be 
displayed comprises a recognition unit (31) for recognizing and 
identifying devices in the environment of the user, and that the 
system is adapted to determine which of the stored control pan- 

25 els to be displayed based on which of the devices is identified. 

7. A system according to claim 6, characterized in that said 
recognition unit (31) is adapted to recognize and identify unique 
Identification markings (9a, 9b) on the devices (11a, lib). 

30 

8. A system according to any of the claims 5-7, characterized 
in that the system is arranged so that It changes the virtual con- 
trol panel displayed when another device is recognized and 
identified, and when the user has accepted the device. 

35 
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9. A system according to any of the previous ciainfts, charac- 
terized in that said portable identification element (3) is adapted 
to be carried by the user during interaction with the virtual con- 
trol panel. 

5 

10. A system according to any of the previous claims, charac- 
terized in that said portable identification element (3) is attach- 
able to the body of the user. 

10 11. A system according to any of the previous claims, charac- 
terized in that said display unit comprises a wearable display 
device (7) showing the user said view. 

12. A method that enables a user to interact with a virtual con- 
15 trol panel using a user controlled pointing object, wherein the 
method comprises: 

- receiving data representing the position of the user con- 

trolled pointing object (1), 

- receiving data representing the position of a portable identi- 
20 fication element (3), 

- storing at least one pre-defined graphical interface repre- 

senting a control panel of a device, 

- generating a graphical representation of the control panel of 

the device based on said pre-defined graphical interface, 
25 - registering said graphical representation of the control 

panel in a fixed relation to said portable identification 
element, based on said data representing the position of 
the identification element, to produce a virtual control 
panel, 

30 - displaying a view comprising the real world and the virtual 

control panel projected in a fixed relation to said portable 
identification element, and 

- performing actions in response to the users interactions 

with the virtual control panel, wherein the actions to be 
35 performed is determined based on the position of said 
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user controlled pointing object in relation to the position 
of the virtual control panel. 

13. A method according to claim 12, comprising modifying the 
5 appearance of the virtual control panel in response to Interac- 
tions between the user controlled pointing object and the virtual 
control panel. 

14. A method according to claim 12 or 13, wherein it comprises, 
10 defining a two-way communication between the virtual control 

panel and the device, sending information to the device regard- 
ing the users actions with the virtual control panel, receiving 
data from the device, generating a graphical representation of 
the received data and displaying the data on the virtual control 
15 panel. 

15. A method according to claim 14, wherein said data is dis- 
played on the virtual control panel in response to interactions 
between the user controlled pointing object and the virtual con- 

20 trol panel. 

16. A method according to any of the claims 12 - 15, wherein 
the method comprises: 

- storing a plurality of pre-defined graphical interfaces, each 
25 representing a control panel of a particular device, 

- determining which of the stored control panels to be dis- 

played, and 

- generating a graphical representation of the control panel to 

be displayed based on the pre-defined graphical inter- 
30 face of the control panel to be displayed. 

17. A method according to any of the claims 12 - 16, wherein at 
least one of the stored graphical Interfaces comprises more than 
one graphical view to be displayed on the virtual control panel, 

35 and which of the views to be displayed is determined based 
upon the users actions. 
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18. A method according to claim 16, wherein it comprises rec- 
ognizing and identifying a device, determining which of the 
stored control panels to be displayed based on the Identified 

5 device, and generating a graphical representation of the control 
panel of the identified device based on the stored graphical in- 
terface of the identified device and displaying a view- comprising 
the real world and the virtual control panel of the identified de- 
vice projected in a fixed relation to said portable Identification 
10 element. 

19. A method according to claim 18 characterized in that each 
device is provided with a unique identification marking and a de- 
vice Is recognized by Identifying Its unique Identification mark- 

15 ing. 

20. A method according to any of the claims 18 - 19, wherein 
the virtual control panel displayed is changed when another de- 
vice Is recognized and identified, and when the user has ac- 

20 cepted the device. 

21. A method according to any of the claims 12-20, wherein 
said portable identification element Is carried by the user during 
interaction with the virtual control panel. 

25 

22. A method according to any of the claims 12-21, wherein 
the virtual control panel comprises virtual interaction members 
and an audio and/or visual feedback is generated when the user 
activates any of the virtual Interaction members. 

30 

23. A computer program comprising program instructions which, 
when loaded into a computer, causes the computer to perform 
the process of any of the claims 12-22. 

35 24. A computer readable medium having a program recorded 
thereon, where the program is to make a computer perform the 
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Steps of any of the claims 12 -22, when said program is run on 
the computer. 



